Efficient Temporal Mean Shift for Activity Recognition in Video

We propose a temporal mean shift algorithm that clusters spatio-temporal regions in video by exploiting the temporal nature of video. Extracting spatio-temporal regions is often one of the first pre-processing steps in an activity recognition system. Our key contribution is the insight that mean shift clustering can exploit the fact that there is typically very little change between successive video frames. Most of the pixels, and therefore the clusters, shift only slightly from frame to frame. Since mean shift is an iterative procedure, fewer iterations are required for convergence if the initial search is already close to the local optimum. Our temporal mean shift algorithm exploits the temporal similarity between successive frames by initializing the search using the modes found in the previous frame. Standard Mean Shift Clustering •Finds local modes in the clusters of points •Iterative gradient ascent procedure •Let fi be a d-dimensional point in a set of n feature points f1...fn. We calculate where j = 1, 2, ..., with kernel g, typically a Gaussian kernel, and bandwidth h. •The mode is the limit of the series, where y(i,j) converges to a fixed point. •The inner loop of mean shift requires a range-search of neighbors fk near y(i,j) , which is particularly time-consuming in high dimensions. •Previous methods in optimizing standard mean shift focused on reducing the time needed for range-search of neighbors. Temporal Mean Shift Clustering •We observe that in most videos, there is very little change between successive frames. Video sequence